feat: support for Spark 4 #589
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…late run up to stackabletech/operator-templating@e3ffe71 (#590) Reference-to: stackabletech/operator-templating@e3ffe71 (chore: SDP 25.7 templating updates)
* feat(helm): Add RBAC rule for automatic cluster domain detection * chore: Bump stackable-operator to 0.94.0 and update other dependencies * chore: Update changelog * chore: Add sparkhistory and shs shortnames
adwk67
requested changes
Jul 28, 2025
|
|
||
| === Maven packages | ||
|
|
||
| The last and most flexible way to provision dependencies is to use the built-in `spark-submit` support for Maven package coordinates. | ||
| The downside of this method is that job dependencies are downloaded every time the job is submitted and this has several implications you must be aware of. | ||
| For example, the job submission time will be longer than with the other methods |
There was a problem hiding this comment.
Suggested change
| For example, the job submission time will be longer than with the other methods | |
| For example, the job submission time will be longer than with the other methods. |
| The downside of this method is that job dependencies are downloaded every time the job is submitted and this has several implications you must be aware of. | ||
| For example, the job submission time will be longer than with the other methods | ||
| Network connectivity problems may lead to job submission failures. | ||
| And finally, not all type of dependencies can be provisioned this way. Most notably, JDBC drivers cannot be provisioned this way since the JVM will only look for them at startup time. |
There was a problem hiding this comment.
Suggested change
| And finally, not all type of dependencies can be provisioned this way. Most notably, JDBC drivers cannot be provisioned this way since the JVM will only look for them at startup time. | |
| And finally, not all type of dependencies can be provisioned this way. | |
| Most notably, JDBC drivers cannot be provisioned this way since the JVM will only look for them at startup time. |
| If you need access to JDBC sources from your Spark application, consider building your own custom Spark image as shown above. | ||
| As mentioned above, not all dependencies can be provisioned this way. | ||
| JDBC drivers are notorious for not being supported by this method but other types of dependencies may also not work. | ||
| If a jar file can be provisioned using it's Maven coordinates or not, depends a lot on the way it is loaded by the JVM. |
There was a problem hiding this comment.
Suggested change
| If a jar file can be provisioned using it's Maven coordinates or not, depends a lot on the way it is loaded by the JVM. | |
| If a jar file can be provisioned using its Maven coordinates or not, depends a lot on the way it is loaded by the JVM. |
| @@ -3,5 +3,6 @@ | |||
| // Stackable Platform documentation. | |||
| // Please sort the versions in descending order (newest first) | |||
|
|
|||
| - 4.0.0 (Hadoop 3.4.1, Scala 2.13, Python 3.11, Java 17) (Experimental) | |||
| - 3.5.5 (Hadoop 3.3.4, Scala 2.12, Python 3.11, Java 17) (Deprecated) | |||
There was a problem hiding this comment.
Suggested change
| - 3.5.5 (Hadoop 3.3.4, Scala 2.12, Python 3.11, Java 17) (Deprecated) | |
| - 3.5.6 (Hadoop 3.3.4, Scala 2.12, Python 3.11, Java 17) (LTS) |
| @@ -3,5 +3,6 @@ | |||
| // Stackable Platform documentation. | |||
| // Please sort the versions in descending order (newest first) | |||
|
|
|||
| - 4.0.0 (Hadoop 3.4.1, Scala 2.13, Python 3.11, Java 17) (Experimental) | |||
| - 3.5.5 (Hadoop 3.3.4, Scala 2.12, Python 3.11, Java 17) (Deprecated) | |||
| - 3.5.6 (Hadoop 3.3.4, Scala 2.12, Python 3.11, Java 17) (LTS) | |||
There was a problem hiding this comment.
Suggested change
| - 3.5.6 (Hadoop 3.3.4, Scala 2.12, Python 3.11, Java 17) (LTS) | |
| - 3.5.5 (Hadoop 3.3.4, Scala 2.12, Python 3.11, Java 17) (Deprecated) |
|
🟢 Local tests (nightly suite) are all good. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Part of: #586
Depends on the corresponding image pr stackabletech/docker-images#1216
Spark 4 is considered experimental because of the following issues:
The integration tests have been updated to exclude spark 4 for the tests known to cause problems.
Definition of Done Checklist
Author
Reviewer
Acceptance
type/deprecationlabel & add to the deprecation scheduletype/experimentallabel & add to the experimental features tracker